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Introduction: Why Engage in Program Evaluation ? 

Evaluations of extended-day programs are of interest to diverse 
groups,** For those underwriting the costs of an extended-day program, 
evaluation can ensure program accountability by demonstrating that a 
program is doing what the director, staff and others claim it should be 
doing. For parents, program evaluation can provide a plan for improving 
the quality of care for their children. From an academic perspective, 
program evaluation offers the opportunity for applied research and testing 
of theories of social behavior. For the practitioner (the program director, 
teacher, supervisor or coordinator), program evaluation makes it possible 
to assess the program that he or she is providing for children, their 
parents, and the co;r>munity. 

Recently, the notion of program evaluation has been incorporated Into 
a wide range of federal, state and local programs. Lawmakers, administra- 
tors, and taxpayers have begun to question the effectiveness of social 
programs which are subsidized with public funds from federal, state or 
local sources. The . idea of accountability has achieved such popularity 
that many states have enacted "sunset" regulations. Such laws require 
that funding be stopped if a program fails to meet a certain number of its 



* The authors with to thank Clifford Baden, Anne Coolidge, Andrea 
Genser, and Gwen Morgan of Wheelock College for all their insightful 
comments and assistance in the refining of this document, 

**Throughout this paper, we will be referring to extended-day programs 
as those programs attended by school-age children, between the ages of 
5 and 14, before and after school. In such programs, children receive 
care for two to eight hours a day in a center-based or family-based 
setting, (Bergstrom & Dreher, 1975), 
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goals within a stated period of time. For many programs that receive 
federal funds, an evaluation component is mandated; other programs take 
the initiative themselves to evaluate the effectiveness of their program 
services to children and families. 

From a more global perspective, program evaluation can also demon- 
strate the usefulness of a program in terms of social reform. Extended- 
day programs, for example, can be viewed as experiments to solve a social 
concern. Through an evaluation of the appropriateness of certain pro- 
grams and curriculum experiences the teaching of basic skills, the 
provision of adequate supervision, the use of specific facilities, and the 
like — the well-being of school-age children who have mothers employed 
outside the home can be ascertained. Variations on a basic program model 
may be seen as socic! axperiments; evaluations can help the public assess 
which ones are more likely to produce the desired outcomes. 

What is Program Evaluation ? 

Program evaluation has been defined by Wholey and others in the 
following terms: 

Evaluation (1) assesses the effectiveness of an ongoing program in 
achieving its objectives; (2) relies on the principles of research 
design to distinguish a program's effects from those of other 
forces working in the community; and (3) aims at program im- 
provement through a modification of current operational practices. 
(Miringoff, 1980, p. 134). 

It is also necessary to note at this point that the evalution should be 
planned so that it feeds back information which can be the basis for im- 
proving the program. This facet of the planning stage is critical. 

Our discussion focuses primarily on a formal approach to evaluation. 
It is assumed that specific funds to conduct the evaluation will be made 
available from the agency or agencies sponsoring the extended-day pro- 
gram. Or from other sources. 

Who Should Be Involved in the Process? 

As discussed by Miringoff, any plan for program evaluation must 
involve full participation of staff members involved in the program. The 
practitioner responsible for program administration, for example, should 
play an integral role in every stage of the evaluation, even though, outside 
evaluators are involved. The practitioner is the natural person to deal 
with outlining the short- and long-term goals of the program, defining the 
types of questions to be asked, deciding who is to be involved in the 
evaluation process, and exploring the possible ways to answer the ques- 
tions. 

Community people should be involved in the evaluation process. They 
often have keen insights into the program, especially if the program chil- 
dren use facilities with which they are associated, such as the library and 
media center or recreation facilities. In fact school-age children are also 
of value to the evaluation process. Often articulate and aware of those 
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aspects of the program that they find favorable or unfavorable, these 
children can make specific suggestions as to how the program could be 
changed. 

By involving themselves, their staff, the children, parents, and other 
community members in data gathering , practitioners are more likely to 
produce information needed to evaluate the achievement of the program's 
goals. Information gathered from many representative sources can help 
the practitioner and other program advocates provide a balance to efforts 
of policy makers and others who may find the programs ineffective or 
undesirable for one reason or another. Also, a well-conceived evaluation 
at the local level may counteract the negative effects of some poorly con- 
ceived large studies, conducted by evaluators who do not have a "grass- 
roots** understanding of extended-day programs. 

What is the Evaluation Process? 

The evaluation process consists of seven steps (see Figure 1). The 
practitioner plays a key role at each step. 



Figure I 

The Evaluation Process 

Step 1 Understand the Fundamental Question: 

Why Evaluate? 

Step 2 Clarify Short- and Long-^Term Program Goals 

Step 3 Plan the Evaluation Design 

Step 4 Develop a Strategy to Collect Information 

Step 5 Analyze the Information Collected 

Step 6 Prepare and Disseminate the Evaluation Report 

Step 7 Use Findings to Improve Program 



Step 1, Understanding and Accepting the Purposes of the Evaluation . The 
process of evaluation is costly, time-consuming and demanding of all those 
involved. The costs and potential benefits to staff, children, parents and 
groups within the community must be identified and weighed before deter- 
mining what specific aspects of the program are to be evaluated and how 
they are to be evaluated. Who will be involved directly in the evaluation 
process? How much time will this involvement require? Will it necessitate 
giving up other responsibilities? Are all interested parties aware of the 
fact that an evaluation wilt be conducted? Do they know why? (For 
example, is the evaluation mandated by agencies outside the program? Is 
it desired by people working inside the program? Who is most interested 
in seeing the evaluation results? Can the practitioner anticipate at the 
outset how he or she will make use of the results?) On balance, is par- 
ticipation in the evaluation process perceived as a positive, healthy, step? 
Once these questions have been answered, a commitment to proceed with 
the evaluation can be made. 
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Step 2. Clarifying Short- and Long-Term Programs , One of the most ef- 
fective ways to clarify goals is to have a practitioner, usually the direc- 
tor, work with a team of people which includes staff, children, parents, 
and community representatives. Ultimately, it will be the responsibility of 
the practitioner to interpret the different needs of children, families, and 
the community, and to express these as clearly stated short-term and 
long-^term program goals- The practitioner then communicates these goals 
to the evaluator(s) . 

This step is critical, and may well prove to be more difficult than it 
sounds. Reviews of several extended-day programs suggest that while 
many have broadly staled objectives, there is usually a lack of specificity 
as to what these mean on a short-term or long-term basis. The dialogue 
represented below (between a practitioner and a program evaluator) illus" 
trates the gap which may exist between the practitioner's general concep- 
tualization of the program and a statement of measurable goals for the 
program. 

Program Evaluator: How would you describe the goals of your 
program? 

Practitioner: As a director of a small extended-day program, I 
am trying to provide a safe and secure place for children; a place 
where several children can be together, work together and enjoy 
themselves in the afternoon, ( am committed to providing a 
recreational program for the children where they can be safe and 
secu re. 

Program Evaluator: Can you describe what you mean by "safe 
and secure" in more specific terms? 

Practitioner: By "safe and secure," I mean a place where chil- 
dren can come to play, engage in activity and be with other 
children and an adult. A place which the children know is al- 
ways going to be there, A place where they can come after 
school is out and be with others- 

Program Evaluator: Does "safe and secure** mean anything in 
terms of staff-child ratio? 

Practitioner: A staff-child ratio of one adult to eight children 
appears to work for us. 

Program Evaluator: Telt me, does "safe and secure" mean any- 
thing in terms of the indoor and outdoor environment for the 
children? Do you have indoor and outdoor facilities? 

Practitioner : Yes! We have beautiful facililities, both indoors 
and outdoors. Both areas have everything one could imagine and 
hope to use. 

Program Evaluator: Explain for me the physical layout of the 
indoors facilities and purpose of each piece of equipment, and 
please do the same for the outdoors. Perhaps we should consider 
doing a diagram. 

Practitioner: I'd be glad to and as I do it the recreational aspect 
of the extended-^day program will become obvious to you. 
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Indoors we have areas where the children can engage in arts 
and craftS; such as weaving, painting and creating three-dimen- 
sional objects. Actually, there are three areas in the large play- 
room which are set up for these types of creative and expressive 
activities. There are other areas for indoor activities also. 

Outdoors we have a number of areas in which the children 
can plan and engage in sports such as soccer, badminton and 
* baseball. In addition to setting up the space and planning for 
these activities indoors and outdoors^ we also have staff special- 
ists who teach and help the children to engage in recreational 
activities; for example, one staff person coaches sports such as 
baseball, soccer and badminton, and another person teaches 
several arts and crafts activities to the children. 

Program Evaluator: Okay, it appears that the challenge for us 
will be first to develop an evaluation process to understand the 
extent to which this is a safe and secure place for the children, 
and second, ^0 determine the extent to which this *s a recrea- 
tional program. A variety of methods for collecting this informa- 
tion might be useful. The perceptions of staff, children, parents 
and others will be critical. 

This dialogue suggests how the practitioner and the evaluator can 
work together towards a better understanding of the program. The dia- 
logue also highlights the importance of the second step: ensuring that the 
evaluation design is based on a clear understanding of the goals of the 
program. 

Step 3. Planning the Evaluation Design , This step is rather technical. 
As such, the expertise of the outside evaluator(s) will be helpful at this 
stage. An evaluation design which depends upon random assignment of 
children to different groups is usually impossible when working with an 
ongoing social program, such as an extended-day program. Therefore, 
some other evaluation design must be devised. This might involve mea- 
surements of behaviors, attitudes, skills, and the like, before and after 
participation in the extended-day program, or the comparison of children 
enrolled in one program with children enrolled in another program. While 
the technical aspects of this step are the responcibility of the evalua- 
tor(s), the practitioner should remain informed, to ensure that the evalua- 
tion design does not impose an unexpected burden on the program and 
does not violate any ethical constraints. 

Step 4, Developing a Strategy to Collect Information . This step obvi- 
ously depends upon successful completion of the preceding steps — specific 
program goals must have been identified and an evaluation design must be 
in place. Again, the technical expertise of the evaluator(s) is important 
here. One "control" that the practitioner needs to be aware of is that 
more than one data collection format should be used. There is a wide 
range of formats for collecting data. Dozens of standardized test instru- 
ments exist, and one or more of these may be appropriate for a particular 
program evaluation. Many of these are described in The Eighth Mental 
Measurements Yearbook , Much use can be made of informal assessment 
techniques -- a few of which are described here briefly, A daily log, a 
format for recording observations, enables the staff member to record both 
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daily plans and thoughts and feelings about what has happened each day 
in the extended-day program. A log kept over a period of time is a 
useful reference source; it might be used, for example^ to pinpoint activ- 
ities which occurred on specific days, 

A daily log, kept by the school-age child, is also a useful evaluation 
tooL By encouraging the child to provide a systematic record of his 
activities, the evaluator can determine the type of activities a child en- 
gages in, the amount of time he/she devotes to an activity, the other 
children and adults involved in the activity, and so on. It is helpful to 
have the child record in the diary in a manner facilitating the coding of 
data along specific dimensions. 

Another method is participant observation. This refers to a method 
of study whereby a staff member actively participates in a group in order 
to learn more about both individual members and the group as a whole. 
The staff member might interact with the group in a variety of ways 
including eating, playing games, sports activities, or craft projects. In 
some instances, the children might know they are being observed; in other 
situations the children will be unaware of the staff person's role as ob- 
server. After each session the participant observer records his impres- 
sions and ideas, either in written or taped form. 

Checklists provide a system for recording the presence or absence of 
certain behaviors. ^ The observer is simply required to give a ^V^s" or 
"no" response on the checklist. (A participant observer miqht use a 
checklist after he/she had left the scene of the observation while a non- 
participant observer could simultaneously observe the children and com- 
plete a checklist.) Since the questions on the checklists are compiled in 
advance, they enable one to rapidly gather a great deal of information 
relating to the child's language, cognitive, perceptual, social and emotional 
development. Checklists allow one to quickly determine the specific books 
the children are reading, the games they are playing, and the other ways 
they are spending their time. 

A sociometric test is a series of questions asking children whom they 
would like to associate with in specific situations. The information gath- 
ered from these questionnaires can be combined to make a sociogram, 
which enables staff to learn which children like to associate with one 
another. Sociometric tests sometimes provide information about the status 
of children within the group; that is, which children the group considers 
desirable as workmates and playmates and which children the group con- 
siders less desirable as workmates and playmates. 

Parent interviews provide the opportunity for the extended-day staff 
and parent to discuss the child, to share information, and to gain a better 
understanding of the child in a number of settings. Children this age can 
also be interviewed and are frequently reliable in the information that they 
report. 

A case study of a child and family often provides useful information 
on the meaning of the program for the families involved. The interviewer 
can, for example, gather information regarding the value of the program 
for the child and his/her family. The focus of the case study will depend 
on the objectives of the extended-day program. 
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Portfolios can be designed to store samples of children's creations and 
written work. Each child coutd have his/her own portfolio. As the child 
creates a significant work, either the original work or a photograph of it 
can be placed in the portfolio. Each sample should be accompanied by an 
index card which includes the date the work was completed^ the time of 
day, the name of the person collecting the work, and notes regarding the 
creation. This work documents not only the child's interests, but also the 
extent to which there has been development in specific areas such as the 
writing of stories and poems and painting and drawing. 

The choice of formats used will depend in part upon the evaluation 
design. The practitioner should remember that information collected as 
part of the evaluation process can also be of great use to th*? extended- 
day staff as they talk with children, as they meet with parents, and as 
they work together to develop the program. 

Step 5, Analyzing the Information Collected . Depending on the formats 
used to gather information, there may be anecdotal data, samples from logs 
or portfolios, or statistical data. While the analysis can be implemented by 
outside evaluator(s), the practitioner has a role m ensuring that the 
information is in fact representative of the program and that conclusions 
drawn from the data correspond to reality. Whatever conclusions the data 
point to be they critical or commendatory they should seem valid to 
those who best know the program. 

Step 6, Preparing and Disseminating the Evaluation Report , This task is 
best accomplished as a collaborative effort. The practitioner can provide 
information relating to the background, organization and goals of the 
program, while the evaluator(3) can describe the evaluation design and the 
results obtained from the data analysis. Both practitioner and evaluator 
should identify the practical, political, and ethical implications of the final 
report. Issues of privacy and confidentiality must be respected. Anyone 
quoted by name in the final report should have an opportunity to verify 
the accuracy of the quotation. If the entire valuation has in fact been a 
collaborative venture, the antaysfs and recommendations in the final report 
should not come as a surprise to anyone. Finally, the tone and content of 
the report should be appropriate for the agency or individuals who initi- 
ated the evaluation. If the report, or portions of it, are going to be 
shared with parents, or funding sources, or others, the report should also 
reflect their contributions and views. 

Step 7, Incorporating Evaluation Findings Into the Ongoing Program, 
Whether the evaluation report is positive, negative, or somewhere in be- 
tween, the process should not stop when the report fs distributed. It is 
the practitioner's responsibility to use the information generated by the 
evaluation to improve or strengthen the existing program. The evaluation 
may, for example, suggest a way to clarify or rethink program goals, or 
fndicate a need for new or different staff development efforts to achieve 
progrzr.i goals. The practitioner may want periodically to repeat some of 
the steps of the formal evaluation process on an informal basis or to assess 
the continued growth and development of the program. 

In one sense, the practitioner or program administrator is always 
evaluating the extended -day program, determining whether things are 
going well and whether children, staff and parents are pleased with the 
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program. The experience of a formal evaluation brings an external per- 
spective to this process. It helps to clarify program goals and enriches 
the practitioner's repertoire of techniques for assessing the effectiveness 
of the program, (f the evaluation process is carried oul in a thoughtful, 
comprehensive, and collaborative fashion, it should result in an improved 
program to meet the needs of children, staff, parents, and the community. 
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